feat(voice): add AssemblyAI streaming STT provider#1912
Open
cjol wants to merge 55 commits into
Open
Conversation
🦋 Changeset detectedLatest commit: f169317 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
stof
reviewed
Jul 10, 2026
Spec for @cloudflare/voice-assemblyai — a type-compliant Streaming v3 STT provider for the Agents voice pipeline, mirroring the Deepgram/Telnyx example. Captures scope (STT-only, baseUrl-configurable for AI Gateway), the v3 protocol mapping, test plan, and deferred work (typed gateway helper, Workers AI self-host, on-prem pharma).
Apply design-review simplifications: - options down to apiKey/model/formatTurns/medical/keyterms/baseUrl/params - add `params` passthrough; drop endOfTurnConfidenceThreshold - hardcode encoding=pcm_s16le and sample_rate=16000 (pipeline-fixed) - drop `region` (use baseUrl for EU/gateway); document URLs in README - cut `prompt` from v1; close() no longer awaits Termination - single test file; add Known Limitations (no onError, model-determined language)
…ence
Verified streaming params against AssemblyAI's live API docs (via MCP):
- lock model to u3-rt-pro (the voice-agent model); drop the `model` option
- auth via Authorization header (raw key, no prefix), NOT a ?token= query param
- drop format_turns (u3-rt-pro finals are always formatted)
- fully typed surface, no passthrough: domain, keyterms, minTurnSilence,
maxTurnSilence, interruptionDelay, continuousPartials, baseUrl
- rename medical:boolean -> domain:"medical-v1"|(string&{})
- hardcode speech_model/sample_rate/encoding; conditional params only when set
- limitations: no onError, language model-determined, single model / 6 languages
Verified against AssemblyAI prompting + streaming API docs: - prompt is a connection param (no UpdateConfiguration round-trip needed); it is the u3-rt-pro differentiator and the language-guidance lever. Doc the "omit for the optimized default prompt" recommendation. - vadThreshold (vad_threshold): VAD silence-confidence threshold for noisy envs. - both appended only when set; language limitation softened (guided via prompt).
language_detection applies to u3-rt-pro (native multilingual code-switching) and returns language_code/language_confidence on Turn events. Since the Cloudflare transcript callbacks are text-only, pair the flag with a provider-specific onLanguageDetected(languageCode, languageConfidence) callback to surface it without changing the shared interface.
- Add a `speechModel` option (defaults to u3-rt-pro) with the `AssemblyAISpeechModel` type and model-aware validation that rejects u3-rt-pro-only params (prompt, continuousPartials, interruptionDelay) on universal-streaming models. - Default min/max turn silence to 400/1280 ms (above the server's 100/1000) so speakers can pause mid-thought without the turn being cut off; callers can still override. - Default continuous_partials on for u3-rt-pro for a live mid-turn transcript. - Log unexpected WebSocket closes (auth/session errors only arrive as close frames) and rewrite wss:// to https:// for Cloudflare's fetch-based upgrade.
- Streamline the intro, usage, options table, and how-it-works sections; document the new `speechModel` option and the 400/1280 ms turn-silence defaults. - Add an "AssemblyAI documentation" section linking the relevant streaming pages (Universal-3 Pro, turn detection & partials, message sequence, streaming API, endpoints & data zones, Medical Mode, keyterms) and a dashboard link for the API key.
Real-time voice agent running entirely in a Durable Object: AssemblyAI u3-rt-pro streaming STT, Workers AI LLM (glm-4.7-flash) with tools, and Workers AI MeloTTS for keyless TTS. Browser mic streams 16 kHz PCM over a plain WebSocket via the useVoiceAgent React hook, with barge-in, streaming responses, and conversation memory. Only ASSEMBLYAI_API_KEY is required.
Update the AssemblyAI streaming provider to the current model and API. Plugin (voice-providers/assemblyai): - Lock speech_model to universal-3-5-pro; drop the speechModel option and the old model-compat assertions (universal-3-5-pro is is_u3_pro server-side, so it supports prompt/agent_context/mode/etc). - Add connection params: mode, agentContext, previousContextNTurns, languageCode, voiceFocus, voiceFocusThreshold, validated against the server's ranges (1500-char caps, threshold requires voiceFocus, interruptionDelay 0-1000, previousContextNTurns 0-100). - Defer turn-silence / partials to mode instead of force-defaulting them. - Add session.updateAgentContext(text): sends an UpdateConfiguration with agent_context mid-session (pre-connect buffering, latest-only, 1500 cap). Framework (@cloudflare/voice): - Add optional TranscriberSession.updateAgentContext?(text) (no-op for providers without context carryover), a forwarder on AudioConnectionManager, and pipeline calls so withVoice feeds each spoken reply/greeting back to the transcriber as conversational context. Also update the example, provider/voice READMEs, and voice docs.
When previousContextNTurns is 0, the server discards agent_context (the carryover window is zero), so AssemblyAISession.updateAgentContext() now early-returns instead of sending an UpdateConfiguration the server would throw away. Contained to the plugin — the framework seam still calls updateAgentContext() unconditionally and the provider decides to no-op.
Rename prose references from "Universal-3.5 Pro Streaming" to the official "Universal 3.5 Pro Realtime" across the provider, example, READMEs, and voice docs. The wire value (speech_model=universal-3-5-pro) is unchanged.
- prompt and keyterms are NOT mutually exclusive on streaming u3.5-pro (no server validator enforces it) — correct the docs and the prompt JSDoc. - Raise MAX_PROMPT_CHARS 1500 -> 1750 to match the current API (connection_parameters.py); applies to prompt, agentContext, and the updateAgentContext() truncation. - Add descriptors for the Prompting/Keyterms, Conversation Context, and Voice Focus documentation links.
The "Endpoints & data zones" link pointed at the streaming API parameter reference; repoint it to /docs/streaming/endpoints-and-data-zones, which actually lists the regional WebSocket hosts the descriptor describes.
AssemblyAI requires 50-1000ms of audio per binary WebSocket message and terminates the session (3007) on the second undersized message. Browser mic chunks are 100ms and pass through untouched, but the Twilio/Telnyx adapters relay 20ms frames, which killed the session almost immediately. Accumulate sub-minimum frames until they cross 1600 bytes (50ms at 16kHz mono s16le), matching AssemblyAI's own LiveKit adapter. A sub-minimum tail is dropped at close().
Say 'server default' for previousContextNTurns instead of a number the docs don't pin down, source the keep-context-concise guidance from the Conversation Context page, and cite session-based billing rather than asserting the mechanics.
…g LLM errors
onCallStart queried cf_voice_messages with raw SQL, which throws 'no such
table' on a brand-new agent instance (the voice mixin creates the table
lazily) and silently killed the greeting. Use getConversationHistory(),
which ensures the schema. Also add a streamText onError logger — the AI
SDK otherwise swallows LLM failures, surfacing them only as an empty
reply ('No response generated').
The voice pipeline warns that textStream can join non-adjacent text parts incorrectly (visible as doubled tokens); the other voice examples already return result.fullStream.
glm-4.7-flash was intermittently hanging ~60s and returning 3043/504 from the inference upstream, leaving the agent silent. gpt-oss-20b (one of the voice-agent example's endorsed models) streams in ~1s and handles the tool-calling demo reliably.
…adapter MeloTTS was intermittently failing with 3043 inference errors, leaving the agent silent after transcription succeeded. The framework's default WorkersAITTS (@cf/deepgram/aura-1) is the same path the other voice examples use, and dropping the custom adapter simplifies the example.
…reply gpt-oss-20b emits reasoning tokens before text; with Workers AI's ~256 default cap, longer reasoning consumed the whole budget and the turn ended with finishReason=length and zero speakable text — surfacing as a silent 'No response generated'. Cap at 2048 and log turn transcripts and LLM finish stats so this failure mode is visible in the terminal.
Tested against the live stack: gpt-oss-20b works but its reasoning phase delays the first spoken token and can starve short caps; glm-4.7-flash was intermittently erroring upstream; llama-3.3-70b leaks tool calls as literal JSON text through workers-ai-provider. llama-4-scout executes tools correctly, emits no reasoning phase, and keeps replies concise.
…demo Replace the generic time/weather assistant with Luna Rossa, a restaurant reservation agent — a real use-case shaped around what the integration demonstrates: terse answers after agent questions (agent_context carryover), venue vocabulary (prompt + keyterms), reservations persisted in the DO's SQLite across calls, and availability/booking/lookup/cancel tools. The LLM is OpenAI gpt-4.1-mini via the AI SDK: every Workers AI model tested had a blocking defect in this multi-tool voice loop (glm-4.7-flash upstream errors, llama tool-call JSON leaks and hallucinated confirmation codes, gpt-oss reasoning-only empty replies, mistral token doubling). Verified end-to-end: 4-turn spoken booking conversation with real tool execution and a database-backed confirmation code.
Add an 'Under the hood' panel to the Luna Rossa client that renders debug_event messages broadcast by the agent: the exact agent_context values sent to AssemblyAI (observed by wrapping the transcriber's updateAgentContext), and every tool call and result from onStepFinish. Makes the integration's invisible machinery — context carryover and database-backed tools — visible during a live call.
Replace WorkersAITTS (aura-1) with a small in-example CartesiaTTS adapter — a natural, low-latency voice and a demonstration of bringing your own TTS vendor to the pipeline via the TTSProvider interface. Requests are serialized through a queue with one retry on 429, since Cartesia plans cap concurrent requests and the pipeline synthesizes sentences in parallel (an unqueued burst silently dropped a sentence mid-reply). With the LLM on OpenAI and TTS on Cartesia, nothing uses the AI binding anymore — drop it from wrangler.jsonc, so local dev needs no Cloudflare login, just the three vendor keys.
…tions Per-sentence synthesis over the REST endpoint made each sentence an independent generation, so the voice's pitch and pacing jumped audibly at sentence seams — Cartesia documents this exact failure mode and its fix: stream sentences of one reply into a shared WebSocket context (continuations), so the model extends one generation instead of restarting. The pipeline starts sentence syntheses eagerly, so the reply's first sentence opens the context and yields all of its audio while later sentences join it and yield nothing; a short idle timer finalizes the context and barge-in cancels it. WS audio is raw PCM, wrapped per-chunk in WAV headers for the client's decodeAudioData path (which also decodes exactly, avoiding MP3 padding at seams). One-shot utterances (the greeting) keep the simpler REST/MP3 path.
…pter Keep the example focused on the AssemblyAI integration: the built-in Workers AI TTS (aura-1) needs no extra vendor key or adapter code. The Cartesia WebSocket-continuations adapter lives in branch history (bb9cd02) as a reference for a future @cloudflare/voice-cartesia provider. MeloTTS was evaluated as the non-Deepgram alternative but currently fails ~half of requests upstream (3043).
Two robustness gaps from the final review: (1) if the socket never connects (e.g. a bad API key), feed() buffered mic audio unboundedly in the Durable Object — now capped at ~30s with a single logged warning; (2) WebSocket.send() throws in the gap between the server closing the socket and the close event firing — sends now route through a guarded helper instead of letting the exception escape into the audio pipeline.
… audio synthesize() read response.arrayBuffer() without checking response.ok, so an error body (e.g. the Workers AI free-tier 429 quota JSON) was sent to the client as audio bytes and failed to decode silently. Log the status and body and return null so the pipeline skips the sentence.
Identify traffic from this provider with AssemblyAI's structured user-agent format (AssemblyAI/1.0 (integration=Cloudflare-Agents)), matching how the official SDKs and other integrations attribute usage.
0281b2d to
8ec361c
Compare
agents
@cloudflare/ai-chat
@cloudflare/codemode
create-think
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
Contributor
Author
|
@dlange-aai I've opened a PR from your branch for the AssemblyAI adapter, and I'm ready to merge it. I'm afraid I've pulled out the example, though, in an effort to start to standardise and simplify our various voice examples. The consolidated example is much more generic (doesn't have a particular scenario like the reservation desk), but lets the user set keywords, context, etc. Let me know if there's some feature that you think we're not showing off enough! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds
@cloudflare/voice-assemblyai, a streaming speech-to-text provider for@cloudflare/voice, based on the contributor branch linked in #1900. Closes #1900.Why
@cloudflare/voicealready supports pluggable STT providers, but users who want AssemblyAI streaming transcription need to write their own adapter.TranscriberSessionrather than coupling it to a specific provider. Existing providers can ignore it.Public API Surface
AssemblyAISTT@cloudflare/voice-assemblyai.AssemblyAISTTOptionsTranscriberSession.updateAgentContextCode Changes
voice-providers/assemblyai, following the existing voice provider package structure.onInterim,onUtterance, andonSpeechStartcallbacks.withVoice.examples/assemblyai-voice-agentreservation-agent demo.